10 research outputs found
Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image using CNN
Collaborative robots are becoming more common on factory floors as well as
regular environments, however, their safety still is not a fully solved issue.
Collision detection does not always perform as expected and collision avoidance
is still an active research area. Collision avoidance works well for fixed
robot-camera setups, however, if they are shifted around, Eye-to-Hand
calibration becomes invalid making it difficult to accurately run many of the
existing collision avoidance algorithms. We approach the problem by presenting
a stand-alone system capable of detecting the robot and estimating its
position, including individual joints, by using a simple 2D colour image as an
input, where no Eye-to-Hand calibration is needed. As an extension of previous
work, a two-stage transfer learning approach is used to re-train a
multi-objective convolutional neural network (CNN) to allow it to be used with
heterogeneous robot arms. Our method is capable of detecting the robot in
real-time and new robot types can be added by having significantly smaller
training datasets compared to the requirements of a fully trained network. We
present data collection approach, the structure of the multi-objective CNN, the
two-stage transfer learning training and test results by using real robots from
Universal Robots, Kuka, and Franka Emika. Eventually, we analyse possible
application areas of our method together with the possible improvements.Comment: 6+n pages, ICRA 2019 submissio
Multi-Objective Convolutional Neural Networks for Robot Localisation and 3D Position Estimation in 2D Camera Images
The field of collaborative robotics and human-robot interaction often focuses
on the prediction of human behaviour, while assuming the information about the
robot setup and configuration being known. This is often the case with fixed
setups, which have all the sensors fixed and calibrated in relation to the rest
of the system. However, it becomes a limiting factor when the system needs to
be reconfigured or moved. We present a deep learning approach, which aims to
solve this issue. Our method learns to identify and precisely localise the
robot in 2D camera images, so having a fixed setup is no longer a requirement
and a camera can be moved. In addition, our approach identifies the robot type
and estimates the 3D position of the robot base in the camera image as well as
3D positions of each of the robot joints. Learning is done by using a
multi-objective convolutional neural network with four previously mentioned
objectives simultaneously using a combined loss function. The multi-objective
approach makes the system more flexible and efficient by reusing some of the
same features and diversifying for each objective in lower layers. A fully
trained system shows promising results in providing an accurate mask of where
the robot is located and an estimate of its base and joint positions in 3D. We
compare the results to our previous approach of using cascaded convolutional
neural networks.Comment: Ubiquitous Robots 2018 Regular paper submissio
Transfer Learning for Unseen Robot Detection and Joint Estimation on a Multi-Objective Convolutional Neural Network
A significant problem of using deep learning techniques is the limited amount
of data available for training. There are some datasets available for the
popular problems like item recognition and classification or self-driving cars,
however, it is very limited for the industrial robotics field. In previous
work, we have trained a multi-objective Convolutional Neural Network (CNN) to
identify the robot body in the image and estimate 3D positions of the joints by
using just a 2D image, but it was limited to a range of robots produced by
Universal Robots (UR). In this work, we extend our method to work with a new
robot arm - Kuka LBR iiwa, which has a significantly different appearance and
an additional joint. However, instead of collecting large datasets once again,
we collect a number of smaller datasets containing a few hundred frames each
and use transfer learning techniques on the CNN trained on UR robots to adapt
it to a new robot having different shapes and visual features. We have proven
that transfer learning is not only applicable in this field, but it requires
smaller well-prepared training datasets, trains significantly faster and
reaches similar accuracy compared to the original method, even improving it on
some aspects.Comment: Regular paper submission to 2018 IEEE International Conference on
Intelligence and Safety Robotics (ISR). Camera Ready pape
Robot Localisation and 3D Position Estimation Using a Free-Moving Camera and Cascaded Convolutional Neural Networks
Many works in collaborative robotics and human-robot interaction focuses on
identifying and predicting human behaviour while considering the information
about the robot itself as given. This can be the case when sensors and the
robot are calibrated in relation to each other and often the reconfiguration of
the system is not possible, or extra manual work is required. We present a deep
learning based approach to remove the constraint of having the need for the
robot and the vision sensor to be fixed and calibrated in relation to each
other. The system learns the visual cues of the robot body and is able to
localise it, as well as estimate the position of robot joints in 3D space by
just using a 2D color image. The method uses a cascaded convolutional neural
network, and we present the structure of the network, describe our own
collected dataset, explain the network training and achieved results. A fully
trained system shows promising results in providing an accurate mask of where
the robot is located and a good estimate of its joints positions in 3D. The
accuracy is not good enough for visual servoing applications yet, however, it
can be sufficient for general safety and some collaborative tasks not requiring
very high precision. The main benefit of our method is the possibility of the
vision sensor to move freely. This allows it to be mounted on moving objects,
for example, a body of the person or a mobile robot working in the same
environment as the robots are operating in.Comment: Submission for IEEE AIM 2018 conference, 7 pages, 7 figures, ROBIN
group, University of Osl
MUNDUS project : MUltimodal neuroprosthesis for daily upper limb support
Background: MUNDUS is an assistive framework for recovering direct interaction capability of severely motor impaired people based on arm reaching and hand functions. It aims at achieving personalization, modularity and maximization of the user’s direct involvement in assistive systems. To this, MUNDUS exploits any residual control of the end-user and can be adapted to the level of severity or to the progression of the disease allowing the user to voluntarily interact with the environment. MUNDUS target pathologies are high-level spinal cord injury (SCI) and neurodegenerative and genetic neuromuscular diseases, such as amyotrophic lateral sclerosis, Friedreich ataxia, and multiple sclerosis (MS). The system can be alternatively driven by residual voluntary muscular activation, head/eye motion, and brain signals. MUNDUS modularly combines an antigravity lightweight and non-cumbersome exoskeleton, closed-loop controlled Neuromuscular Electrical Stimulation for arm and hand motion, and potentially a motorized hand orthosis, for grasping interactive objects.
Methods: The definition of the requirements and of the interaction tasks were designed by a focus group with experts and a questionnaire with 36 potential end-users. Five end-users (3 SCI and 2 MS) tested the system in the configuration suitable to their specific level of impairment. They performed two exemplary tasks: reaching different points in the working volume and drinking. Three experts evaluated over a 3-level score (from 0, unsuccessful, to 2, completely functional) the execution of each assisted sub-action.
Results: The functionality of all modules has been successfully demonstrated. User’s intention was detected with a 100% success. Averaging all subjects and tasks, the minimum evaluation score obtained was 1.13 ± 0.99 for the release of the handle during the drinking task, whilst all the other sub-actions achieved a mean value above 1.6. All users, but one, subjectively perceived the usefulness of the assistance and could easily control the system. Donning time ranged from 6 to 65 minutes, scaled on the configuration complexity.
Conclusions: The MUNDUS platform provides functional assistance to daily life activities; the modules integration depends on the user’s need, the functionality of the system have been demonstrated for all the possible configurations, and preliminary assessment of usability and acceptance is promising
Joint human detection from static and mobile cameras
Efficient pedestrian detection is a key aspect of many intelligent vehicles. In this context, vision-based detection has increased in popularity. Algorithms proposed often consider that the camera is mobile (on board a vehicle) or static (mounted on infrastructure). In contrast, we consider a pedestrian detection approach that uses information from mobile and static cameras jointly. Assuming that the vehicle (on which the mobile camera is mounted) contains some sort of localization capability, combining information from the mobile camera with the static camera yields significantly improved detection rates. These sources are fairly independent, with substantially different illumination and view-angle perspectives, bringing more statistical diversity than a multicamera network observing an area of interest, for example. The proposed method finds applicability in industrial environments, where industrial vehicle localization is becoming increasingly popular. We implemented and tested the system on an automated industrial vehicle, considering both manned and autonomous operations. We present a thorough discussion on practical issues (resolution, lighting, subject pose, etc.) related to human detection in the scenario considered. Experiments illustrate the improved results of the joint detection compared with traditional independent static and mobile detection approaches
Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image Using CNN
Collaborative robots are becoming more common on factory floors as well as regular environments, however, their safety still is not a fully solved issue. Collision detection does not always perform as expected and collision avoidance is still an active research area. Collision avoidance works well for fixed robot-camera setups, however, if they are shifted around, Eye-to-Hand calibration becomes invalid making it difficult to accurately run many of the existing collision avoidance algorithms. We approach the problem by presenting a stand-alone system capable of detecting the robot and estimating its position, including individual joints, by using a simple 2D colour image as an input, where no Eye-to-Hand calibration is needed. As an extension of previous work, a two-stage transfer learning approach is used to re-train a multi-objective convolutional neural network (CNN) to allow it to be used with heterogeneous robot arms. Our method is capable of detecting the robot in real-time and new robot types can be added by having significantly smaller training datasets compared to the requirements of a fully trained network. We present data collection approach, the structure of the multi-objective CNN, the two-stage transfer learning training and test results by using real robots from Universal Robots, Kuka, and Franka Emika. Eventually, we analyse possible application areas of our method together with the possible improvements
Transfer Learning for Unseen Robot Detection and Joint Estimation on a Multi-Objective Convolutional Neural Network
A significant problem of using deep learning techniques is the limited amount of data available for training. There are some datasets available for the popular problems like item recognition and classification or self-driving cars, however, it is very limited for the industrial robotics field. In previous work, we have trained a multi-objective Convolutional Neural Network (CNN) to identify the robot body in the image and estimate 3D positions of the joints by using just a 2D image, but it was limited to a range of robots produced by Universal Robots (UR). In this work, we extend our method to work with a new robot arm - Kuka LBR iiwa, which has a significantly different appearance and an additional joint. However, instead of collecting large datasets once again, we collect a number of smaller datasets containing a few hundred frames each and use transfer learning techniques on the CNN trained on UR robots to adapt it to a new robot having different shapes and visual features. We have proven that transfer learning is not only applicable in this field, but it requires smaller well-prepared training datasets, trains significantly faster and reaches similar accuracy compared to the original method, even improving it on some aspects